1. Field of Invention
This invention relates to computer input and output devices, specifically to those which provide force feedback, and to those which can be used as a pointing device for graphical user interfaces.
2. Description of Prior Art
Computers are becoming increasingly important as a productivity tool. They continue to improve dramatically in terms of computational speed, memory, storage and display. However, the interface between users and the computer has not changed significantly since the introduction of the mouse and the graphical user interface. The human-computer interface must be improved for users to increase their productivity and take better advantage of the new capabilities computers provide.
Many common computer interface operations are best performed with a direct manipulation interface. For example, when using a drawing application, it is easier for the user to point at the object they wish to select, rather than use a voice recognition interface in which they must describe the object they wish to select.
Typically, direct manipulation interfaces combine a high-resolution pointing device, used to move a cursor on the screen, with some way to initiate an action at the current location. For example, a mouse may employ rotary optical encoders to measure the distance moved, and one or more buttons for xe2x80x9cclickingxe2x80x9d on the object beneath the cursor (e.g., selecting, actuating, dragging, or otherwise manipulating an on-screen object.).
While this was a significant improvement over previous devices, such an interface does not come close to fully exploiting the abilities people have to manipulate objects with their hands. Existing devices have one or more of the following drawbacks:
No Direct Mapping Between the Hand and the Display
Direct mapping is used herein to describe the case where a one-to-one correspondence exists between the position of a cursor on a screen and the position of a user""s hand, and also implies that there is a unique hand position for every cursor position. Input devices which do not move, such as trackballs, joysticks, the IBM TrackPoint(trademark) and the Synaptics TouchPad, lack such a direct mapping. No matter where the cursor is, the user""s hand is in essentially the same location. A mouse also lacks a direct mapping, for at least two reasons. First, there is a non-linear relationship between the speed of the mouse and the speed of the cursor on the screen. This results in a different position depending on how quickly the mouse is moved from one location to another. Second, the mouse is often picked up and moved during use, particularly if the working area is limited.
Direct mapping is important because it better leverages a user""s spatial skills. Humans have a keen sense of the position of their hands in relationship to their body and their environment. Taking advantage of these spatial skills is valuable because the cognitive load placed on the user by the computer interface is decreased, leaving the user""s attention available for performing work. For example, when dragging an object from one point on the screen to another, a user must pay close attention to a cursor""s position and look for visual feedback indicating the cursor is positioned properly, in order to manipulate an on-screen object. During this process, the user""s attention is not available for other tasks (e.g., reviewing files, program output, and the like). Some existing input devices have a direct mapping between the hand and the screen, such as touch screens and digitizing tablets. These devices suffer from other infirmities, as described below.
Lack of Dynamic Haptic Feedback
Haptic feedback is a preferable characteristic for input devices. The term haptic feedback as used herein means communicating information to a user through forces applied to the user""s body. Typically, the position of some portion of an input device changes along at least one degree of freedom depending on the force applied by the user. For example, when pressing a button on a mouse, the button does not move until the applied force reaches a certain threshold, at which point the button moves downward with relative ease and then stops (e.g., the sensation of xe2x80x9cclickingxe2x80x9d a button). The change in the position of the button communicates to the user through their sense of touch that the mouse click was successful. Note that a device with haptic feedback can be an input device (initiating an action) and an output device (giving haptic feedback indicating that the action was initiated) simultaneously.
Input devices that are completely devoid of haptic feedback, such as membrane keyboards and touch screens, have not gained widespread acceptance for desktop computers as a result of this deficiency. Thus when using such input devices, users are uncertain whether a finger press was registered by the computer and so must pay special attention to visual or auditory feedback to get this confirmation. This decreases data entry rates, making users less productive and the computer interface less enjoyable to use.
Mice, trackballs, joysticks, and other devices often provide buttons for initiating actions that provide haptic feedback. For example, the stylus used with a graphics tablet has a spring in its tip so the position of the pen relative to the tablet can vary depending on the applied force. However, such devices have the same haptic response regardless of the state of the user interface. For example, if a user clicks the mouse on a graphical button that is disabled, the haptic response of the mouse button is no different from that of clicking a button that is enabled, and so is misleading to the user because no action will result from the click. What is needed is an input device which provides dynamic haptic feedback. Haptic feedback is termed herein as being dynamic to indicate that the haptic feedback can be altered over time (e.g. by means a software application) in order to provide additional information to a user.
A number of devices having dynamic force feedback exist. Most of these lack a direct mapping between the hand and the device (e.g. force-feedback joysticks). Others have a direct mapping but are primarily designed for use in three-dimensional applications such as virtual reality or tele-operation. Most productive work done on computers is two-dimensional in nature, such as spreadsheets and page layout. These productivity applications would not enjoy significant benefits from the use of a three-dimensional input device. These devices have additional drawbacks, as outlined below.
User Interaction is Encumbered or Impeded
Many input devices encumber the user by requiring them to move at least a portion of the input device during use. For example, the time it takes to move the cursor across the screen with a mouse is increased because the user must accelerate and decelerate the mass of the mouse, in addition to the mass of their hand. Other input devices do not add inertia but impede the user in other ways. With a trackball, for example, multiple sweeping motions are required to move the cursor large distances, which is awkward and time consuming. With a joystick, for example, the force applied relates to the speed of the cursor on the screen, which may require the user to wait when the cursor is moving relatively large distances.
Any input device which must be located and/or manipulated before use suffers from such problems to at least a certain extent (e.g., mice and some force reflecting interfaces, among others). For example, if a person not currently using a computer and wants to press a graphical button on computer""s display, they must find and grasp the mouse, move the mouse to position the cursor over the button, and then click the button. In contrast, a touch screen leaves the user unencumbered. They can reach out and press a graphical button on the display directly, with no intermediate steps. A touch screen, however, suffers from the previously-described infirmity of lacking haptic feedback.
Insufficient Support for Multiple Interactions
Most input devices, such as the mouse, trackball, joystick, the Synaptics TouchPad and the IBM TrackPoint(trademark), only support a single interaction at a time. However, people have two hands which they are innately able to use together. Two single-interaction devices have been combined to provide two points of control, but confusion can arise because the correspondence between screen cursors and pointing devices is not apparent. Because these devices lack a direct mapping to the screen, their physical positions cannot resolve the correspondence between an input device and its cursor. Moreover, no provision is made for the interaction of multiple users. With a single input device, only a single user may xe2x80x9cownxe2x80x9d the device at any given time, and (given a single input device) users must take turns interacting with the computer. This is obviously a cumbersome and awkward technique when multiple users wish to work collaboratively on a given project.
Embodiments of the present invention overcomes conventional limitations by providing a device having a direct mapping, for example, between the touching portion of a user""s hand and the position of a cursor on a display and an output in the form of dynamic haptic feedback, without encumbering or impeding the user and allowing a large number of simultaneous interactions. The device provides direct mapping to reduce the conscious effort required for relatively pedestrian tasks such as interacting with a graphical user interface (GUI). The user""s interaction with the device is not hampered by a need to laterally move any portion of the device.
The device provides dynamic haptic feedback. Haptic feedback is termed herein as being dynamic to indicate that the haptic feedback can be altered over time (e.g. by means a software application) in order to provide additional information to a user. In the previous example, a disabled button would have a different feel from that of an enabled button, allowing a user to discern that a graphical button was not enabled, using their sense of touch. The device also supports multiple interactions. Having more than two points of control is useful when multiple users collaborate at the same computer. Allowing a large number of interactions at once allows multiple users to interact with the computer simultaneously. Another benefit of having more than two points of control is the ability of a user to employ multiple fingers for pointing purposes, even in combination.
Embodiments of the present invention take the form of an input and output device for a processor. In one embodiment, an input/output device has a horizontal two-dimensional area which can be touched simultaneously (e.g., with the hands) in multiple places. The location of each touch is measured and the area surrounding each touch moves vertically and provides dynamic haptic feedback to the user. The device has a control processor that communicates with another processor on which software applications are executed. The control processor continually sends the current attributes of all touches in progress, and receives commands which specify the type of haptic response each touch should exhibit.
The touchable area is comprised of a grid of haptic elements, referred to herein as haptels. Haptel is used herein to describe a haptic feedback device with linear motion having a touchable surface substantially perpendicular to the direction of motion. A haptic feedback device is used herein to describe an input and output device with a moving portion manipulated by a user, one or more sensors that measure the position and/or various derivatives of position and/or the forces applied to the moving portion, one or more effectors which can apply forces to the moving portion, and a processor which measures the sensors, computes a response, and drives the effectors to create a range of haptic effects.
In one embodiment, each haptel includes a position sensor to measure the vertical position of the surface within its range of travel, an electromagnetic linear actuator to provide a controllable vertical bi-directional feedback force, and a touch location sensor to measure the coordinates of a single touch within its bounds. Preferably, the haptel grid is covered by a single sheet of flexible material that protects the haptels and hides the grid from view.
The haptels have their sensors and effectors interfaced to a control processor. The control processor measures the position of haptel surfaces and allows information such as velocity, acceleration, and applied force to be derived. Alternatively, sensors can be included in each haptel to provide such measurements (and others) directly. The control processor computes the desired feedback force for each haptel and drives the actuators to generate the appropriate forces. The haptic response of each haptel may be configured to be essentially arbitrary within a certain range. The range of available effects depends on the type of sensors employed, the bandwidth and precision of the sensors and effectors, the resolution of the analog-to-digital and digital-to-analog conversion performed, the amount of available processing power and the update frequency of the control loop, among other factors. These tradeoffs would be apparent to one skilled in the art of force feedback design.
Because the touchable area is comprised of many haptels, each of which can function independently, the device allows multiple touches at once. Each haptel responds to only one touch at a time, so that there is a lower bound on the distance between two touches which do not interfere with each other. The worst-case value of this minimum distance is approximately the diagonal size of a haptel. However, in a specific instance the minimum distance can be substantially smaller depending on the locations of the two touches. Smaller haptels allow touches to be closer to one another.
A typical interaction is a user pressing a graphical button displayed as part of a GUI. The finger touches the device, landing on a specific haptel. The overall location of the touch is determined by the touch location sensor of the haptel in combination with the location of that haptel within the haptel grid. The touch location is communicated to a processor (e.g., a computer) which discovers that a graphical button is xe2x80x9cunderneathxe2x80x9d the touch, and therefore communicates this information to the control processor to use a xe2x80x9cbuttonxe2x80x9d haptic response for this touch. As the user presses down on the haptel, the control processor responds with a feedback force which increases as the surface is depressed until the position reaches a certain threshold, at which point the feedback force is quickly reduced. This causes the applied force to momentarily exceed the feedback force, which results in the quick downward movement of the haptel surface. In this way a xe2x80x9cclickingxe2x80x9d sensation is conveyed to the user. Preferably, the computer is continually informed of the state of the touch so that when the haptel reaches the bottom of its travel, the computer executes the action represented by the graphical button and displays the button in its activated state.
If the graphical button is disabled, the computer has the control processor use a xe2x80x9cdisabled buttonxe2x80x9d haptic response. In this response the feedback force increases with position at a higher rate than the xe2x80x9cbuttonxe2x80x9d response with no force drop-off. This creates the sensation of an unyielding surface which informs the user than the action represented by the graphical button cannot be initiated.
The preceding descriptions assume that each touch falls within the bounds of a single haptel, but this need not be the case. If the touchable area of the device is mapped to a GUI in which interface elements can be placed anywhere, some will happen to be located on the edge between two haptels or the vertex where four haptels meet. A touch on such a control is therefore likely land on more than one haptel. Such xe2x80x9cborder touchesxe2x80x9d can be transparently handled by the device. The first step is to merge related touches. If two touches appear simultaneously on adjacent haptels a short distance apart, the device can safely infer that the touches are really a single touch on the border between those two haptels. Similar inferences can be made for touches that appear simultaneously near the vertex of any number of haptels.
Once the set of haptels is determined, the haptels are managed in a coordinated fashion. The center of the touch is computed, preferably by weighting each touch location by the force applied to that haptel, and then dividing by the total force applied to the haptels involved. Likewise, the collective surface position, velocity, and acceleration are computed, preferably by weighted average of the haptels involved. Other weightings are possible, including equal weighting of values. The applied force measurements of the haptels involved may be summed to compute the total force applied. The haptic response is then computed from these collective measurements in much the same way they would be computed for a single haptel, resulting in a collective feedback force. This feedback force is distributed across the haptels involved in the touch in proportion to the amount of the total applied force lands on each haptel. In addition, a restoring force pulls the haptels towards the collective position to prevent surfaces from drifting apart due to measurement errors and other factors. As a result, the total feedback force is effectively distributed across the haptels involved in the touch, and the haptel""s surfaces will have similar position, velocity, and acceleration. This provides the illusion that a single surface was pressed, making the coordinated nature of the touch undetectable by the user.
Not only can such device coordinate a fixed set of haptels, but it can also transparently add and remove haptels from the coordination set over time. This is necessary during xe2x80x9cdraggingxe2x80x9d operations in which touches move across the device. When a touch gets close to another haptel, the newly-added haptel is added to the coordination set. This has the effect of causing its surface to become flush with the haptels already involved in the touch. Preferably, this is done without affecting the feel of the touch in progress. When the touch moves far enough away from a given haptel, that haptel is removed from the coordination set, leaving it free to participate in another touch.
This coordination effectively makes the haptels"" gridded nature invisible to the user and to software applications. The computer specifies the response for a touch in a declarative fashion, and the device ensures that this response will be generated regardless of where the touch falls, how many haptels are involved in the touch, or whether the touch moves. Device-specific information provided to the computer might include the minimum allowed distance between independent touches, so that the computer can separate controls designed for simultaneous use appropriately or give feedback to the user when one touch ventures too close to another.
The foregoing is a summary and thus contains, by necessity, simplifications, generalizations and omissions of detail; consequently, those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the present invention, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below.